Dual Oculometer System for Aviation Crew Assessment 
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Oculometers are useful tools for ascertaining the manner in which pilots deploy visual attentional 
resources, and for assessing the degree to which stimuli capture attention exogenously. The aim of this 
effort was to obtain oculometer data comfortably, unobtrusively, reliably and with good spatial resolution 
over a standard commercial aircraft flight deck for both individuals in two-crew operations. We installed 
two remote, five-camera SmartEye systems in the Integration Flight Deck (IFD) simulation facility, 
replicating a modern commercial aircraft. We present here the results of validation exercises, lessons 
learned for improving data quality, and initial thoughts on the use of paired oculometer data to reflect 
crew workload, coordination, and situation awareness, in the aggregate. We conclude with a description 
of future work to improve this installation and extend oculometer capabilities to other simulation 
environments at NASA Langley. 


Oculometers 

Knowing where someone is looking is very useful in 
assessing their interaction with artifacts and their 
environment. As such, oculometers have been 
developed for eye tracking since the late 1800 ’s/early 
1900s ( e.g ., Delabarre, 1898; Dodge & Cline, 1901; Judd 
et al ., 1905), and the history of their development is 
interesting (see Jacob & Karn, 2003). Early systems 
were impractical for naturalistic investigations; requiring 
participants’ head to be motionless, or having elements 
of the apparatus on the participant’s eye. Systems 
evolved towards this end, and became head-mounted, 
which made data collection more robust, but early 
implementations could be uncomfortable, and required 
special integration with other necessary equipment 
(headsets, helmets). 

Image processing advances and digital technology 
provided the necessary technology to make significant 
advances in eye tracking capabilities and their use in less 
clinical settings. Head-borne systems became much 
more compact, robust, and comfortable; but they are still 
subject to artifacts associated with changes in 
registration once seated on the head. Further, most head- 
borne systems that are smaller and more comfortable do 
not provide head position data, and so are not able to 
provide true xyz coordinate data in the context of the 
operator’s environment. Remote camera systems have 
the advantage of allowing participants to operate 
completely naturalistically, and free of encumbrances. 


These systems illuminate the eye with infrared pulsed 
lights and strategically placed cameras obtain images of 
the head and eye. Images of the participant’s face are 
taken from the perspectives of the different cameras. 
After the researcher identifies and marks features (for 
example, the inside comer of the eyes) on each image, 
the system creates a model of the head and eyes. Using 
assumptions of eye geometry and the environment’s 
geometry, a calibration procedure in which the subject is 
asked to gaze at points in the environment allows the 
system to map identified eye positions to locations in the 
external space. 

Oculometers designs include dark pupil, bright pupil, 
and recently those that can operate in both modes; 
typically bright pupil when able, but resorting to dark 
pupil when necessary. Bright pupils are obtained by 
having the illumination source coaxial to the optical 
path, which illuminates the retina and causes essentially 
“red eye.” Bright pupil eye tracking is more robust in 
varying light levels, but is not appropriate for outdoors 
as the additional infrared from sunlight will interfere 
with system response. When the illumination source is 
not coaxial, the pupil is dark. Dark pupil systems rely 
on identifying the edge of the pupil from the start of the 
iris, and so can be difficult with light eyes (which appear 
dark when illuminated with infrared light). Both 
systems rely on the detection of pupil center with respect 
to a “comeal reflection”, a spot of light that corresponds 
to the light source as reflected off the cornea. 



Advances in the collection and interpretation of 
oculometer data are as important as the technology for 
obtaining gaze and head position data. Initially, and for 
some time, the data obtained by oculometers was simply 
the image of the eye moving. Advances in computational 
power allowed real-time analysis of oculometer data and 
the superimposition of point of gaze on the external 
scene available to subjects. More recently, oculometer 
data has been fused with other data streams to better 
characterize the interplay between a participant’s 
attention deployment and environmental demands. 

SmartEye Pro 5.5 Eye Tracking System 

The SmartEye system has several setup parameters 
which must be met to yield robust eye tracking results. 
Each SmartEye system can utilize from two through six 
cameras to calculate both head position and eye gaze 
vectors. 

The system calibrates itself with the real world using a 
checkerboard placed at a world model origin from which 
every user defined plane is specified. Each camera must 
be positioned so the checkerboard is clearly visible in 
each camera field of view, limiting the camera 
placement array to 180 degrees or less. The 
checkerboard origin is also the center of the subject head 
box, which is the intersection of all of the system 
cameras’ field of view, within which the system can 
actively track a subjects head. To improve the accuracy 
of the gaze vectors, the system utilizes corneal reflection 
eye tracking, requiring the illuminators to shine on each 
of the subject’s eyes producing a glint that can be 
detected and tracked by the system. 

The NASA Integration Flight Deck Simulator 

We installed the SmartEye oculometer in NASA 
Langley’s Integration Flight Deck (IFD) simulation 
facility. The IFD generally simulates a modern 
commercial transport aircraft and provides researchers 
with a full-mission capability. The cab includes standard 
flight instrumentation and pilot controls, including the 
overhead subsystem panels. The collimated out-the- 
window (OTW) scene is produced by a Rockwell 
EP1000 graphics system, and provides approximately 
200 degrees horizontal by 40 degrees vertical field-of- 
view at 26 pixels per degree. A FMS/CDU is located on 
the center console, and Electronic Flight Bags can be 
placed outboard of the instrument panels for each pilot. 

Installation 

The dual-operator, five-camera oculometer system 
installation was optimized to capture each pilot’s eye 
gaze over a 90+ degree forward looking span region, 10 


degrees above the horizon and 45 degrees below the 
horizon. 

An eye-point was specified for this installation. 
Commercial aircraft are designed to have a pilot eye- 
point and pilots are taught to use sights and heuristics to 
achieve and maintain this eyepoint. An eye-point 
sighting device was mounted low on the glare shield 
such that each pilot could achieve the appropriate eye- 
point. This eyepoint location was the focal point for all 
installed cameras as well as the origin for the subject’s 
head box and world coordinate model of the IFD. 

Cameras were located such that at least two cameras 
could capture a crew member’s eyes when they looked at 
the areas of interest. These areas of interest included: 
the instrument panel, the control display unit for the 
flight management (CDU/FMS) system, the mode 
control panel, and the forward view out the window. As 
a result, camera locations for each side of the flightdeck 
are: inboard of the CDU/FMS, inboard of primary flight 
displays on the mid-instrument panel, on the center post 
of the window, under the glareshield and above 
instrument panel, and on the outboard glareshield wing 
panel (Figure 1). The initial configuration used only 
8mm lenses, but the final configuration used a 
combination of lens lengths (6, 8, and 12mm) to achieve 
best performance. Flash locations were constrained 
relative to the locations of cameras; they can be no 
closer than 10 cm from a lens, and must be placed such 
that as a set, they illuminate the face for all cameras 
(Figure 1). Cameras, except the two in the instrument 
panel, have a nearby flash. 

There were several constraints and requirements that 
dictated the locations of cameras and flashes. While we 
aimed to achieve best oculometer performance for the 
viewing regions of interest, and to make the installation 
unobtrusive, we were constrained by a requirement that 
existing instrumentation and structures behind the panel 
not be compromised. In part due to these constraints, 
each camera could not have a paired, co-axial flash 
which means that the resulting implementation is a dark 
pupil system. 




Figure 1. Camera & Flash Locations on the Pilot Side 
(which are mirrored on the left side of the flightdeck). 

Validation Metrics 

Head tracking quality (as calculated by the SmartEye 
system) for each of these points, is 100% for good 
quality tracking from at least two cameras, 50% for 
tracking with only one camera, and 0% for no tracking at 
all. Only a value of 100% is considered acceptable. 

Eye tracking quality (ability of system to calculate a 
gaze vector, as recorded by the system) varied across the 
matrix of look points. Gaze quality values (as calculated 
by the SmartEye system) of 75% (normalized over time) 
were considered to be of good quality and 50% or less 
were considered of poor quality gaze vectors. 

In addition to assessing head and eye tracking quality 
(which describes whether the system reported a good 
lock on features), we also assessed the spatial accuracy 
of the eye gaze vectors; that is, does the gaze vector 
accurately and precisely indicate the intended target of 
the participant’s focus. Spatial accuracy was calculated 
by seeing if the gaze vector intersections were consistent 
with those points in the world coordinate model of the 
environment when participants were asked to look at 
certain calibration points. 

Lab Validation Results 

Initial testing was conducted on a scale replica of the 
IFD cockpit with three participants. 

The initial IFD replica configuration was 100% capable 
of tracking two subjects’ head positions over all 
specified regions of interest and included nearly 90% of 
the entire forward looking field of view. The subject 
with glasses was difficult to track in the inboard region 
of the out-the- window view because glasses’ frames 
interfered with the camera’s view. 


Eye tracking quality in the IFD replica was better than 
75% for most of the instrument panel, the forward out- 
the-window view, and the CDU region of the center 
console for initial participants without glasses. Eye 
tracking performance for the participant with glasses 
degraded in the regions just under the glareshield and the 
center instrument panel displays. In all subjects, the 
view OTW to the outboard side was not well tracked. 

Spatial accuracies of less than l°of visual angle were 
considered good, 1° through 2° acceptable, and greater 
than 2° considered poor (Figure 6). 



Figure 2. Spatial accuracy of subject gaze across flight 
panel instruments (Green = <1°, Yellow = <2°) 


For reference, if a pilot is sitting at a normal distance of 
33” from the primary flight display, Figure 7 shows the 
size of the gaze vector spatial distribution error that 1° 
and 2° of visual angle represents. 
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Figure 3. Visual angle depiction. 


Experiment Validation Results 

The lab validation was used to test locations of cameras 
and flashes. This process resulted in design 
requirements for the actual IFD installation (Figure 1). 
The first experiment conducted with this installation 



provided an opportunity to obtain validation data in the 
context of actual use during flight operations and 
procedure use, and with a wider variety of individuals. 

For this study, 22 commercial airline pilots flew the IFD 
simulator in two-crew operations in the busy, terminal 
area surrounding Boston Logan airport. Crews were 
exposed to four experimental conditions (voice 
communication only, Data Comm only, Data Comm 
with Moving Map Display, Data Comm with Moving 
Map displaying taxi route), and each condition was used 
to during both an arrival and a departure scenario. Data 
Comm conditions required the crew to respond to an 
uplinked message via the CDU located on the center 
console. Data from all experimental conditions were 
used for validation of oculometer performance. 

In summary, the results we obtained during the study 
were acceptable for the purpose of supporting the 
experiment, but were not as good as obtained in our 
initial lab configuration. Figure 4 shows the mean and 
standard deviations for head and eye tracking quality, 
averaged over session duration, for both Pilots Flying 
(PF) and Pilots Monitoring (PM). 



Figure 4. Bar charts for head & eye quality, PF & PM 

Because we encountered significant differences between 
the results of participants with and without glasses in the 
lab validation, we wanted to see how experimental 
subjects with and without glasses faired in the 
experimental validation. Figure 5 indicates that pilots 
without glasses yielded higher average gaze quality than 
pilots with glasses; however, differences were minimal 
with less than 10% difference between the two groups. 



Figure 5. Bar chart for eye quality glasses/no glasses. 

To evaluate the accuracy of data while looking in the 
areas of interest for this study, we assessed the average 
eye tracking quality when the system reported hits in 
these AOIs. Figures 5 and 6 show the averaged eye gaze 
quality for PF and PM, respectively, for each AOI. 



Figure 6. PF Eye Quality for AOIs 


Data required to assess spatial accuracy for the 
experimental sessions was not available for analysis 

Lessons Learned 

Variance in performance can be attributed to several 
factors, including camera coverage variation across the 
flight deck, individual differences in facial features, 
camera calibration error, and profile generation errors. 
All of these factors can be calculated and mitigated 
except for camera coverage which is corrected by the 
installation. 
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Figure 7. PM Eye Quality for AOIs 

Cameras and flashes were placed according to the 
requirement that at least two cameras have a good image 
of the eyes for all points of interest. While the lab 
assessment provided a good estimate for placement, we 
realized later that the participants were being more 
restrictive in their head movements than subjects were in 
the experiment. This was particularly true for the PMs; 
who interacted more with the CDU and would 
sometimes lean down when interacting with this device. 
While the lab participants attempted to replicate these 
actions, it was clear to us that the actual participants’ 
movements were, in contrast, not constrained by 
thoughts of system performance, and more naturally 
varying. 

For the SmartEye system, eye tracking quality is 
dependent on the subject remaining in the defined head 
box (the intersection of the cameras’ fields of view). For 
typical PF tasks (left seat), the initially designed head 
box was sufficient. However PMs, in the right seat, 
moved much more to accomplish their tasks, and so 
tended to exceed the boundaries of the defined head box 
region. One solution would be to increase the head box 
region, but this effectively reduces the resolution of the 
images and therefore negatively affects eye tracking. To 
at least assure accurate head tracking of the PM, and at 
the sacrifice of some eye tracking quality, smaller focal 
length lenses were used on some right side cameras. 
This effectively increased the size of the head box for 
the PM. This allows researchers to track at least head 
movements of the PM consistently, while still 
maintaining an acceptable level of eye tracking quality 
when s/he was in the head box. 

In the experimental validation, we encountered a broader 
set of facial characteristics. Certain facial characteristics 
are difficult (e.g. droopy eyelids, squinty eyes) but by 


far, glasses presented the most difficulty. SmartEye 
flashes created reflections on some glasses which could 
interfere with feature detection for eye comers. We have 
since learned that computer glare coatings tend to cause 
problems. In addition, glasses could occlude some of 
the facial features differently for different camera views. 

One method for alleviating this problem is to mark 
visually distinct features of the glasses as an “other” 
feature in the facial profile. This method improved the 
success of tracking subjects with glasses, and highlights 
the fact that selecting the right features for the SmartEye 
to discern eye and head position is a large part of 
obtaining good eye and head tracking. Good features are 
those with high contrast; those that are clearly visible in 
at least 3 of the camera views; and which, as a set, span 
the operator’s face. One feature commonly used is the 
earlobe. In the aviation context, pilots often wear 
headsets, which cover this feature. We affixed a label to 
the headset center and used this as an “other” feature. 
While this is not a perfect solution, since the label can 
move relative to the head, pilots typically find a 
comfortable position for a headset and this position 
appeared to be relatively stable over the 12-17 minute 
mns we observed in the experimental validation. 

Future Work 

The goal of this installation is to provide NASA Langley 
researchers with a built-in capability to sensitively and 
unobtmsively assess attention allocation in novel 
avionics and procedure designs, and to be able to assess 
the impact of these not only on single pilot performance 
and workload, but also on the crew as a unit. We aim to 
develop oculometric measures of individual workload, 
and measures that characterize crew situation awareness 
and workload. This is the subject of ongoing 
dissertation research following in the vein of earlier 
work on individual workload modeling (Ellis & Schnell, 
2009). We are also investigating methods by which to 
improve the rapid characterization of oculometer data 
quality. 

Currently, the data resulting from this initial installation 
resides in two separate SmartEye systems, and is 
collected with a common GPS timestamp that is also 
provided on simulation aircraft parameter datafiles and 
event files. Future work will include the coordination of 
these data. One approach we are investigating is the 
University of Iowa’s Cognitive Aviation Toolkit 
(Schnell, Keller & Macuda, 2007) which will allow the 
integration of simulation state data, images from the 
simulation displays, point of gaze data, and 
physiological data. This system would need to be 
extended to accommodate a second stream of gaze data. 


Work is also underway to not only improve the IFD 
installation, but to extend oculometer capabilities to two 
other simulation cabs at NASA Langley; the Research 
Flight Deck (RFD) and the Generic Flight Deck (GFD). 
The RFD Simulator is an advanced all-glass jet transport 
simulator. The GFD is an advanced all-glass generic 
simulator, which can be used to represent a wide range 
of vehicle types including advanced transports and 
spacecraft. In addition, we aim to ruggedize these 
installations such that they can be used in these 
simulation cabs while they are on a motion-base 
platform. 
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